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FOREWORD 


The  Battlefield  Information  Systems  Technical  Area  of  the  U.  S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  is  concerned,  in  part,  with  human  information  analysis, 
interpretation  functions,  and  subsequent  utilization  of  the  products  in  intelligence  systems. 

The  objective  is  to  provide  technological  advances  in  human/machine-  aided  tactical  intelligence 
information  processing  and  the  translation  of  these  advances  in  support  of  intelligence  systems 
requirement  and  design  decisions,  and  formulation  of  doctrine  and  procedures. 

Achievement  of  the  required  technology  is  often  inhibited  by  a lack  of  understanding  of 
fundamental  principles.  Where  this  is  true,  the  requirement  is  to  increase  the  scientific  basis 
underlying  the  state  of  the  art. 

One  such  area  is  the  assessment  of  subjective  value  or  worth,  which  has  been  addressed  in  ARI 
Technical  Paper  254,  "Techniques  for  the  Assessment  of  Worth."  A functional  assessment 
technology  is  needed  to  support  efforts  such  as  determination  of  the  value  of  intelligence  data  and 
improvement  of  intelligence  collection  procedures. 

The  present  publication  presents  a strategy  for  selecting  the  most  appropriate  worth  assessment 
technique  for  a given  situation.  The  research  effort  is  responsive  to  requirements  of  RDTE  Project 
20062I0IA754.  "Intelligence  Information  Processing,"  and  to  special  requirements  of  the  U.  S. 
Army  Intelligence  Center  and  School. 

ARI  research  in  this  area  is  conducted  as  an  in-house  research  effort  augmented  by  contracts 
with  organizations  selected  as  having  unique  capabilities  and  facilities  for  research  in  a specific 
area.  The  present  study  was  conducted  jointly  by  personnel  of  the  Army  Research  Institute  and 
the  Industrial  Engineering  Department  of  the  University  of  Wisconsin  at  Madison. 


"J.  E.  UHLANER 
Technical  Director 
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A STRATEGY  FOR  SELECTING  A WORTH  ASSESSMENT  TECHNIQUE 

BRIEF 


Requirement 


To  determine  the  factors  influencing  the  outcome  of  worth  assessment  and  to  develop  a 
strategy  for  selecting  the  most  appropriate  worth  assessment  technique  for  a specific 
decisionmaking  problem. 


Procedure 

A large  number  of  techniques  of  worth  assessment  a term  applied  to  the  analysis  of  subjective 
value  judgments  and  their  reduction  to  a quantitative  scale  • have  been  developed.  In  the  present 
analysis,  a strategy  was  developed  for  selecting  a worth  assessment  technique  for  specific 
application  based  on  a 3-round  Delphi  procedure. 

In  the  first  round.  41  factors  influencing  the  selection  and  successful  use  of  worth  assessment 
techniques  were  identified  by  a questionnaire  distributed  to  analysts  active  in  the  worth 
assessment  field. 

The  second  Delphi  questionnaire  was  used  to  determine  the  perceived  relative  importance  of 
the  factors.  Ten  composite  factors,  four  dichotomous  and  six  continuous,  appeared  to  account  for 
the  41  factors  and  their  relative  weights. 

Reference  profiles  for  nine  techniques  were  developed  for  the  six  continuous  factors,  based  on 
a third  Delphi  questionnaire. 


Findings 

The  research  product  is  a three-step  strategy  for  selecting  a worth  assessment  technique  for  a 
given  situation,  based  on  the  ten  composite  factors. 

The  first  step  is  the  analysis  of  the  four  dichotomous  factors  to  screen  out  techniques  not 
suited  to  the  situation.  The  second  step  is  an  evaluation  of  the  situat  .mui  <i<.|uiiements  (attributes) 
for  a technique,  based  on  the  six  continuous  factors 

Finally,  the  situation  profile  is  • oinpaiest  with  the  reference  profiles  to  determine  which  of  the 
nine  techniques  it  most  approimatr  ts»i  the  given  situation. 
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Utilization  of  Finding* 

Worth  assessment  is  a vital  ingredient  of  advanced  intelligence  information  processing.  The 
strategy  developed  cannot  be  said  to  insure  selection  of  the  perfect  technique.  However,  such 
strategy  should  serve  to  increase  likelihood  of  success  in  worth  assessment  application  by  leading 
the  user  to  consider  situational  factors  and  attributes  of  available  worth  assessment  technioues. 
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A STRATEGY  FOR  SELECTING  A WORTH  ASSESSMENT  TECHNIQUE 


Subjective  worth  forms  the  basis  for  the  selection  of  future  alter- 
natives and  the  evaluation  of  past  actions.  Techniques  for  measuring 
subjective  worth  extend  back  to  the  eighteenth  century.* 


In  the  last  70  years,  there  has  been  an  explosive  growth  in  the 
research  on  the  theory  of  subjective  worth  and  a proliferation  of  assess- 
ment or  scaling  techniques  in  this  field.  This  Technical  Paper 
addresses  two  related  questions. 


First,  what  specific  factors  are  likely  to  affect  the  success  of  a 
worth  assessment  technique?  Second,  what  strategy  can  be  used  to  select 
an  appropriate  worth  assessment  technique  for  a specific  application? 


There  are  factors  which  not  only  differentiate  one  worth  assessment 
technique  from  another,  but  also  determine  the  relative  success  of 
techniques  in  a specific  application.  An  extensive  literature  search 
yielded  little  information  on  why  a particular  technique  was  used  in  a 
given  situation  or  how  the  technique  was  chosen.** 


The  idea  that  one  technique  would  be  more  appropriate  in  a given 
situation  while  another  would  be  better  in  a different  situation  has 
not  been  extensively  considered. 


METHOD 


Given  the  paucity  of  information  in  the  literature,  the  remaining 
source  of  information  was  the  collective  knowledge  of  the  practitioners 
and  experts  in  the  field  of  worth  assessment,  gathered  by  a Delphi 
procedure.  The  Delphi  procedure  involves  the  sequential  individual 
questioning  usually  by  questionnaire)  of  a set  of  experts,  interspersed 
with  information  and  opinion  feedback  derived  from  earlier  parts  of  a 


questionnaire  program. 


“Edwards,  W,  The  Theory  of  Decision  Making.  Psychological  Bulletin, 

1954,  51(4),  380-417. 


“"Kneppreth,  N.  P.,  Gustafson,  D.  H.,  Leifer,  R.  P.,  and  Johnson,  E.  M. 
Techniques  for  the  Assessment  of  Worth.  ARI  Technical  Paper  2^4  , 
August  1074 . AD  784  620) 


“"“Dalkey,  N. , and  Helmer,  0.  An  Experimental  Application  of  the  Delphi 
Method  to  the  Use  of  Experts.  Management  Science,  1Q63,  ,2.  4 58-4  b7 . 


In  the  present  project,  a three-round  Delphi  procedure  was  us^d  to 
systematically  solicit  and  collate  informed  Judgments  on  factors 
influencing  the  selection  of  a worth  assessment  technique. 

To  keep  the  studv  within  reasonable  bounds,  nine  worth  assessment 
techniques  were  selected  for  participant  consideration.  Fach  of  the 
techniques  has  several  variations  and  each  represents  a group  of 
techniques  with  common  attributes. 

A brief  description  of  each  technique  is  given  in  Figure  1,  while 
more  complete  descriptions  are  available  in  Kneppreth  et  al , 1974.  In 
the  present  discussion,  the  role  of  worth  assessor  is  differentiated 
from  the  role  of  decis ionmaker  (DM). 

Although  in  many  cases  the  worth  assessor  and  the  DM  are  the  same 
person,  the  two  roles  represent  different  perspectives  of  a worth  assess- 
ment problem. 

The  analysis  of  a worth  assessment  problem  to  select  a technique  for 
a given  evaluation  is  the  role  of  an  assessor/analyst.  The  evaluation 
of  the  worth  of  a specific  level  of  an  attribute  is  the  DM's  role.  Thus, 
from  the  assessor/analvs t ' s viewpoint,  the  characteristics  of  the  DM 
are  factors  in  a worth  assessment  problem. 

In  the  following  sections,  "factor"  and  "attribute"  have  slightly 
different  meanings.  Factor  refers  to  a characteristic  or  peculiarity 
of  the  entire  decision  environment.  Attribute  is  used  in  the  more 
restrictive  sense,  referring  to  characteristics  of  variables  being 
assessed  - in  this  case,  the  assessment  techniques.  An  attribute  is  a 
factor  by  this  definition,  but  a factor  is  not  necessarily  an  attribute. 


THE  DFLPHI  PROCEDURE 

A three-round  Delphi  was  used  to  obtain  the  informed  judgments  and 
opinions  of  48  experts  and  practitioners  in  the  field  of  worth 
assessment . 

Round  1 

The  first  Delphi  questionnaire  was  sent  to  48  people  and  was  designed 
to  determine  what  factors  were  important  in  choosing  a worth  assessment 
technique.  The  48  persons  in  the  sample  were  asked  to  respond  to  the 
following  two  questions: 
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What  factors  would  you  consider  important  in  deciding 
which  worth  assessment  technique  to  use  and  why? 

What  specific  factors  would  likely  affect  the  success  of  a 
worth  assessment  session?  Give  an  example  of  what  you  mean. 

Responses  were  received  from  16  individuals  who  collectively  listed  41 
different  factors  (Figure  2).  Anything  considered  important  by  a 

respondent  was  included  as  a factor. 

/ 

Factors  include  characteristics  of  several  widely  different  areas: 
the  assessor,  the  problem,  the  decisionmaker,  and  the  techniques.  The 
diversity  of  opinion  and  the  number  of  factors  identified  underscore  the 
complexity  of  the  worth  assessment  process. 


Round  2 

The  second  Delphi  questionnaire,  sent  to  the  same  48  people  who 
received  Round  1,  was  designed  to  determine  relative  importance  of  the 
factors  identified  in  Round  1.  Individuals  were  provided  a list  of  the 
41  factors  separated  into  five  categories.  They  were  asked  to: 

Choose  the  nine  most  important  factors  from  the  entire  list 
and  indicate  (in  the  space  for  comments)  why  you  feel  these 
are  important,  and  how  they  affect  selection. 

Rank  order  the  nine  factors,  assigning  a value  of  "1"  to 


the  most  important  and 


to  the  least  important. 


Individuals  were  also  asked  to  indicate  the  two  or  three  most  important 
factors  within  each  of  the  five  categories.  Responses  were  received 
from  23  individuals,  including  14  of  the  respondents  to  the  first 
questionnaire . 

The  relative  importance  of  each  factor  was  determined  by  adding  the 
inverse  rankings  that  each  factor  received  and  then  multiplying  this  sum 
by  the  number  of  respondents  who  indicated  the  factor  was  important. 

The  relative  weights  normalized  over  all  41  factors  on  a 0 (least  impor- 
tant) to  100  (most  important)  scale  are  shown  in  Figure  2. 

The  4 factors  in  Figure  2 with  a 0 relative  weight  were  not 
considered  critical  by  any  respondent.  The  37  critical  factors  can  be 
useful  in  themselves  to  the  practitioner  as  they  elaborate  what  individ- 
uals in  the  field  consider  important  when  assessing  worth.  The  knowl- 
edge that  certain  factors  may  adversely  affect  the  outcome  should  help 
the  practitioner  anticipate  and  alleviate  problems  encountered  in  an 
assessment  session. 

The  diversity  of  factors  considered  important  indicates  the  need  for 
further  research  in  the  area.  Many  of  the  factors  identified  are  inter- 
dependent, and  their  empirical  relationship  to  assessment  success  is 
unknown . 
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It  would  be  useful  to  know  the  exact  dependencies  among  factors  and 
the  exact  relationships  between  the  factors  and  assessment  success.  This 
information  would  increase  not  only  the  likelihood  of  successful  worth 
assessment,  but  would  enable  the  development  of  more  precise  selection 
and  assessment  strategies. 

Round  3 

llie  procedure  was  based  on  the  assumption  that  there  are  attributes 
which  differentiate  various  worth  assessment  techniques  in  the  context 
of  a particular  decision  situation.  The  factors  identified  in  Round 
1 are  not  satisfactory  in  this  regard. 

While  some  of  the  factors  do  relate  to  attributes  of  assessment 
techniques,  they  do  not  differentiate  between  techniques.  Many  relate 
to  more  than  one  attribute,  and  several  do  not  relate  at  all  to  attri- 
butes of  the  techniques. 

Therefore,  a set  of  attributes  was  inferred  to  account  for  the 
concerns  expressed  in  the  first  two  rounds  of  the  Delphi  and  to  differ- 
entiate among  the  techniques.  By  successively  partitioning  the  •» 1 
factors  Figure  2),  six  continuous  and  four  dichotomous  technique 
attributes  appeared  most  appropriate  Figure  *>)  . 

These  ten  attributes  appear  to -account  for  nearly  all  the  factors 
listed  in  Figure  2,  with  the  principal  exception  of  several  relating  to 
the  competence  of  the  assessor.  These  ten  technique  attributes  are  the 
basis  of  the  selection  strategy  developed  later.  While  this  set  of 
attributes  is  not  unique,  it  accounts  for  selection  differences  stated 
in  the  Delphi  responses. 

The  four  dichotomous  attributes  are  easy  to  evaluate,  as  the  level 
appropriate  for  each  of  the  nine  techniques  Figure  l)  is  fairly  obvious. 
The  continuous  attributes  are  more  difficult  to  evaluate  and  were  the 
subject  of  Delphi  Round 

The  third  Delphi  questionnaire  was  designed  to  develop  a profile  of 
the  six  continuous  attributes  for  each  of  the  nine  techniques.  The 
questionnaire  was  sent  to  17  individuals,  chosen  for  their  expertise 
and  their  familiarity  with  all  nine  techniques.  Respondents  were  asked 
to  rate  each  method  on  each  of  the  continuous  attributes.  For  instance, 
in  the  case  of  "Assessment  Time"  they  were  asked  to: 
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Please  rate  the  following  techniques  on  the  relative  time 
it  takes  to  complete  an  assessment  of  a given  attribute  with 
a decisionmaker  trained  in  using  the  technique.  This  is  a 
measure  of  the  face-to-face  time  an  assessor  and  decision- 
maker would  spend  in  assessing  a given  attribute.  It  does 
not  include  training  time  of  the  decisionmaker.  Please 
place  a 10  next  to  the  technique  that  you  feel  takes  the 
least  amount  of  time,  and  a 90  next  to  the  technique  which 
would  take  the  most  time.  Rate  the  other  techniques  in 
relation  to  these  two. 

Responses  were  received  from  nine  individuals  and  the  resulting  profiles 

are  given  in  Figures  6 through  14. 


A STRATEGY  FOR  SELECTING  A WORTH  ASSESSMENT  TECHNIQUE 

The  10  decision  situation  attributes  developed  in  the  Delphi 
(Figure  3)  form  the  basis  of  a strategv  for  selecting  a worth  assessment 
technique.  The  four  dichotomous  attributes,  summarized  in  Table  1, 
represent  characteristics  of  the  decision  attributes  in  the  assessment 
problem. 

The  six  continuous  attributes  summarized  in  the  reference  profiles 
(Figures  6-14)  represent  characteristics  of  the  setting  of  the  worth 
assessment  problem. 

The  proposed  selection  strategv  is  designed  to  guide  the  user 
through  an  analysis  of  the  decision  problem  in  terms  of  these  10  attri- 
butes and  to  insure  that  these  factors,  which  might  determine  the 
success  of  the  application  of  worth  assessment,  are  considered  by  the 
user  (Figure  4). 

There  are  three  steps  in  the  proposed  strategy.  The  first  step 
(Block  1 in  Figure  4)  is  the  analysis  of  the  dichotomous  attributes 
to  screen  out  techniques  not  suitable  for  the  situation. 

As  an  example,  if  one  of  the  attributes  to  be  assessed  in  the 
decision  is  discrete  and  cannot  be  approximated  as  continuous,  this 
difficulty  will  rule  out  several  of  the  techniques:  the  graphic, 
constant  probability,  and  indifference  curve  methods. 

The  four  dichotomous  attributes  are  analvzed  using  a series  of 
questions  for  the  assessor/analyst . After  each  question,  one  or  more 
techniques  may  be  ruled  out  from  further  consideration.  In  some  cases 
only  one  feasible  technique  will  remain  at  the  end  of  this  step,  and 
the  analyst  need  go  no  further. 
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In  other  cases,  it  will  be  apparent  that  more  than  one  technique  will 
be  needed.  For  instance,  one  technique  may  be  needed  for  dependent 
decision  attributes,  another  for  the  remaining  attributes. 


If  feasible  to  use  more  than  one  assessment  technique,  subsets  of 
similar  decision  attributes  should  be  handled  separately  in  the  remaining 
two  steps  of  the  selection  strategy. 


The  second  step  (Block  2 of  Figure  4)  is  to  determine  what  values  a 
technique  should  have  for  the  six  continuous  attributes . The  different 
factors  in  the  situation  that  define  the  required  levels  of  each  of  the 
continuous  attributes  are  considered. 

For  instance,  the  time  pressure  of  the  study,  the  amount  of  time 
the  decisionmaker  has  available,  and  the  number  of  attributes  involved 
will  interact  to  define  how  quick  the  technique  chosen  must  be.  Each 
of  the  six  attributes  is  a weighted  composite  of  the  situational 
factors  developed  in  the  Delphi  that  influence  the  value  of  the 
attribute. 

This  step  yields  a profile  of  the  situation  and  defines  a minimally 
acceptable  technique.  By  comparing  this  situation  profile  with  the 
profile  of  each  of  the  techniques,  it  will  be  possible  to  reduce  the 
subset  of  feasible  techniques  still  further. 

In  the  third  step  (Block  3 of  Figure  4),  the  profile  of  the  situation 
is  compared  with  the  normative  reference  profiles  of  each  of  the 
techniques  remaining  from  the  first  step.  Techniques  which  match  or 
exceed  the  minimally  acceptable  profile  on  each  attribute  are  considered 
feasible. 

From  these  techniques,  the  one  that  best  satisfies  other  criteria 
not  yet  considered  in  the  analysis  can  be  selected.  An  example  of  other 
criteria  would  be  the  amount  of  faith  that  the  assessor /analyst  has  in 
the  technique. 

The  main  strength  of  the  strategy  is  not  that  it  insures  a choice 
of  the  "perfect"  technique.  The  chief  strength  is  rather  that  it  guides 
an  analysis  of  the  decision  situation  to  force  consideration  of  factors 
important  in  the  choice  of  a technique. 

All  too  often,  the  choice  of  an  assessment  technique  is  made  on  only 
one  or  two  factors.  Even  then,  the  factors  considered  are  not  always 
the  most  critical.  In  addition,  the  strategy  forces  the  assessor  to 
consider  all  nine  of  the  techniques  rather  than  some  restricted  subset. 
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PROCEDURE  FOR  SELECTING  A WORTH  ASSESSMENT  TECHNIQUE 

Step  1 


The  four  dichotomous  attributes  are  listed  below  as  a series  of 
questions.  If  the  answer  to  any  one  of  them  is  "no,"  go  on  to  the  next 
questior;  if  the  answer  is  "yes,"  continue  with  the  explanations 
following  the  question. 


Qu-tLi  ftOH  1.  Are  any  of  the  decision  attributes  dependent? 

If  the  value  of  an  attribute  depends  on  the  level  of  another  attri- 
bute, then  the  two  attributes  are  said  to  be  dependent.  Stated  another 
way,  if  the  decisionmaker  cannot  specify  values  for  an  attribute  with- 
out knowing  the  value  of  another  attribute  then  dependencies  are 
probably  present. 

The  indifference  curve  method  is  the  only  method  frequently  used  to 
handle  dependent  factors.  This  method  could  be  the  immediate  choice  for 
assessing  dependent  variables.  However,  the  method  has  several  short- 
comings which  often  make  it  advantageous  to  modify  the  situation  and  use 
a different  method.  A series  of  procedures  may  make  possible  the  use  of 
another  method. 

Can  the  dependent  attributes  be  combined  into  a single  factor? 
Combining  decision  attributes  may  enable  the  composite  attribute  to  be 
treated  as  independent.  For  instance,  in  a decision  to  purchase  a car, 
fuel  capacity  (gallons)  and  fuel  economy  (miles /gallon)  might  be 
dependent  attributes. 

Combining  them  mathematically  by  multiplication  give9  a new  attri- 
bute, fuel  range  (miles /tankful) , that  can  be  assessed  as  a single 
independent  attribute.  If  possible  to  combine  dependent  attributes  in 
the  situation  at  hand,  combine  them  and  go  to  question  2. 

Can  the  dependent  attributes  be  assessed  as  pairs,  triplets,  or 
other  combinations?  If  the  set  of  all  meaningful  combinations  of  (fit- 
dependent  attributes  is  of  reasonable  length,  then  ft  mav  be  advantageous 
to  assess  the  attribute  pairs  as  a single  attribute. 

If  color  and  make  of  car  are  dependent  and,  for  instance,  blue  is 
considered  the  best  color  tot  a Ford  and  green  the  be9t  color  for  a 
Chevrolet,  and  three  colors  are  being  considered  for  each  make  of  car, 
then  the  six  color  make  combinations  mav  be  assessed  as  a single 
attribute.  ii  this  combination  is  feasible  in  the  situation,  then  go  to 
question  2. 
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Can  independence  be  assumed?  Even  if  the  attributes  are  dependent, 
it  might  be  useful  to  assume  they  are  independent.  If  the  dependencies 
are  such  that  independence  is  not  an  unreasonable  assumption,  e.g.,  the 
assumption  will  not  affect  the  decision  to  any  great  degree,  then  simplv 
assume  they  are  independent  and  go  to  question  2. 

If  the  decision  attributes  are  dependent,  and  none  of  the  above 
procedures  is  possible,  then  the  indifference  curve  method  must  be 
employed.  However,  in  practice  this  technique  is  limited  to  the  assess- 
ment of  two  continuous  monotonic  attributes  at  a time.  If  more  than  two 
attributes  are  dependent  pairwise,  then  some  more  sophisticated  technique 
than  the  nine  considered  here  has  to  be  used. 

For  the  situation  with  only  pairwise  dependent  attributes,  the 
indifference  curve  method  must  be  used,  and  the  analvst  need  go  no 
further.  However,  if  one  or  more  interdependent  pairs,  plus  one  or  more 
independent  attributes  are  in  the  situation,  continue  to  question  2. 


Question  1.  Does  the  decision  situation  involve  risk? 

Some  analysts  feel  that  a decisionmaker's  sub eri  ; , it  iters 
between  risky  and  riskless  situations.  Him-  . derision  alterna- 
tive is  selected  and  the  outcome  is  ■ i . -one  analysts  prefer  to 

use  an  assessment  technique  that  - some  assessment  of  risk.  If 

the  situation  involves  rf  i . i ben  the  two  gamble  methods  should  be 
favorably  considered  >>•.,!  i,  i ert  ed  only  if  they  clearly  do  not  meet  the 
Other  r r i t e r i > 


N'e.  1.  Are  anv  of  the  attributes  nonmonotonicallv  related  to  value? 

An  attribute  is  nonmonotonicallv  related  to  value  when  an  increase 
in  the  attribute  over  one  part  of  the  attribute's  range  increases  the 
value  to  the  decisionmaker,  but  an  increase  over  another  part  of  the 
attribute's  range  decreases  the  value  to  the  decisionmaker.  A nonmono- 
tonic curve  has  one  Dr  more  peaks  or  humps. 

If  any  attribute  is  suspected  of  being  nonmonotonicallv  related  to 
value,  then  it  is  advisable  to  limit  the  use  of  the  gamble  techniques 
and  to  eliminate  the  indifference  curve  method  for  consideration.  The 
indifference  curve  method  requires  monotonicitv,  and  problems  in 
constructing  wagers  may  arise  with  the  gamble  methods. 
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Question  4.  Are  any  of  the  decision  attributes  discrete? 


If  any  factors  are  discrete  and  cannot  be  approximated  by  a contin- 
uous function,  then  the  graphic  method,  indifference  curve  method,  and 
constant  probability  wagers  should  be  rejected. 

The  dichotomous  attributes  of  the  nine  assessment  techniques  are 
summarized  in  Table  1. 


At  this  point,  the  analyst  should  have  an  indication  of  how  many 
different  techniques  might  be  required  in  the  particular  situation.  For 
instance,  it  might  be  known  that  both  the  indifference  curve  method  for 
the  dependent  attributes  and  one  other  technique  to  handle  discrete 
factors  would  be  necessary  in  the  assessment  process. 

Each  subset  of  attributes  must  be  analyzed  separately  in  steps  2 and 
3.  One  technique,  the  indifference  curve  method,  is  known.  The  rest 
of  the  process  would  therefore  be  needed  only  to  determine  the  best 
technique  for  the  discrete  attributes. 

If  there  are  time  constraints,  some  of  the  subsets  might  be  combined 
at  this  point  to  reduce  the  time  needed  for  training  decisionmakers  in 
several  different  techniques.  Training  can  take  a great  deal  of  time  and 
place  a heavy  burden  on  the  decisionmaker.  If  there  is  only  one  contin- 
uous attribute,  it  may  be  useful  to  assess  it  by  the  same  method  selected 
for  the  discrete  attributes. 

Based  on  the  critical  factors  in  a decision  situation,  questionnaires 
were  developed  (Round  3 of  the  Delphi)  to  evaluate  the  situational 
requirements  for  a worth  assessment  technique  on  the  six  continuous 
factors . 

The  Evaluation  Questionnaires  are  shown  in  the  appendix.  The 
resulting  values  of  these  continuous  factors  obtained  from  the  question- 
naires form  a profile  of  the  minimally  acceptable  worth  assessment 
technique  which  matches  the  decision  situation.  The  same  six  evaluation 
questionnaires  should  be  completed  by  the  analyst  in  order  to  develop  a 
meaningful  comparison  profile  of  the  particular  situation. 
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Table  1. 

Dichotomous  Attributes 

of  Assessment 

Techniques 

Me  thods 

Dependence 

Risk 

Nonmonotonicity  Discreteness 

GAMBLE  METHODS 

Constant 

Probability 

? 

YES 

? 

NO 

Variable 

Probability 

? 

YES 

? 

YES 

MULTIVARIATE  METHOD 

Indifference 

Curve 

YES 

NO 

NO 

NO 

ORDINAL  METHODS 

Ranking 

? 

NO 

YES 

YES 

Pair  Comparisons 

? 

NO 

YES 

YES 

Equivalence 

Grouping 

? 

NO 

YES 

YES 

DIRECT  METHODS 

Double  Anchor 

? 

NO 

YES 

YES 

Single  Anchor 

? 

NO 

YES 

YES 

Graphical 

? 

NO 

YES 

NO 

Note.  The  entries  represent  answers  to  the  questions:  Will  the  technique  handle  dependent  attributes;  does  the  method  involve 
risk;  will  the  method  handle  attributes  nonmonotonically  related  to  worth;  and,  will  the  method  handle  discrete 
attributes. 
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To  illustrate  the  profile  of  a minimally  acceptable  assessment 
technique,  transfer  the  total  scores  developed  from  the  questionnaii.es 
to  the  scales  shown  in  Figure  5.  Draw  vertical  lines  at  the  points 
corresponding  to  the  total  scores,  and  shade  the  portion  of  the  scale 
to  the  left  of  these  marks.  In  this  wav,  the  situation  profile  can 
easily  be  compared  with  the  graphic  reference  profiles  of  the  nine 
assessment  methods  (Figures  6-14) . 


Step  3 

Figures  6-14  give  the  reference  profiles  of  the  nine  worth  assessment 
techniques  considered  in  the  present  development . The  shaded  portion 
on  the  left  of  each  bar  on  the  profiles  extends  to  the  mean  value 
assessed  during  the  Delphi,  and  constitutes  the  acceptable  portion  of 
the  scale. 

The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean,  and  gives  an  indication  of  the  relative  agreement  among  the 
judges.  In  this  step,  the  situation  profile  developed  in  Step  2 is 
compared  to  these  profiles,  and  the  most  suitable  technique  is  chosen. 
This  step  in  the  strategy  is  divided  into  three  parts. 

P<Vit  1 ofi  StZp  3.  The  output  of  the  comparison  part  of  Step  3 will 
be  a subset  of  techniques  that  matches  or  exceeds  the  situation  profile 
on  all  factors.  This  subset  is  referred  to  as  the  feasible  subset  and 
may  contain  no  techniques,  or  one  or  more  techniques.  If  any  techniques 
were  rejected  in  Step  1,  disregard  these  techniques  throughout  this  step. 

Compare  the  situation  profile  with  the  profiles  in  Figures  6 through 
14.  In  each  comparison,  the  technique  should  be  rejected  if  it  does  not 
match  or  exceed  the  situation  profile  on  all  six  factors.  If  the  tech- 
nique matches  or  exceeds  the  profile  for  all  factors,  the  technique  is 
considered  part  of  the  feasible  subset.  Only  the  shaded  part  of  the 
profile  is  considered  in  the  comparisons. 

When  this  process  is  completed,  the  feasible  subset  msv  contain  only 
one  technique:  This  technique  is  the  one  considered  most  appropriate  by 
the  proposed  strategy  for  the  decision  process  under  study,  and  the 
formal  strategy  ends  here.  However,  the  analyst  should  go  back  and 
check  the  selection  against  intuition.  In  some  cases,  there  may  be  no 
techniques  in  the  feasible  subset,  all  the  techniques  having  at  least 
one  factor  which  does  not  match  or  exceed  the  theoretical  profile.  In 
that  case,  further  analysis  is  necessary.  Omit  part  2 and  continue  with 
part  3 of  the  step.  If  the  feasible  subset  contains  more  than  one 
technique,  go  to  part  2. 
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2 5 top  3.  This  part  is  for  selecting  from  a group  of 

feasible  techniques.  If  the  feasible  subset  contains  several  techniques, 
the  final  choice  of  technique  must  be  made  on  the  basis  of  additional 
criteria,  since  any  one  of  the  feasible  techniques  in  the  subset  is 
probably  appropriate  for  the  situation.  As  an  example  of  such  criteria, 
the  analyst  might  choose  the  technique  that: 

1.  he  has  the  most  faith  in  and  is  most  familiar  with,  or 

2.  will  provide  the  DM  with  the  best  feedback,  or 

3.  the  DM  has  used  before,  or 

4.  allows  for  easy  experimenting  and  value  changing  hv  the  DM. 

Possible  criteria  are  numerous,  and  the  choice  can  be  made  on  anv 
criteria  that  the  assessor  deems  important.  If  no  additional  criteria 
are  deemed  important,  and  if  a choice  still  cannot  be  made,  part  3 of 
Step  3 ciin  be  followed  to  make  a final  selection. 

PflAf  3 0|$  Sfep  3.  A method  of  choosing  a technique  on  the  basis  of 
a weighted  aggregate  model  is  appropriate  in  situations  in  which  either 
no  feasible  technique  was  found  in  part  1 of  Step  3 or  no  choice  among 
feasible  techniques  could  be  made. 

First,  the  method  attributes  must  be  rated  to  determine  their  rela- 
tive importance  in  the  given  decision  situation.  Figure  15  gives  direc- 
tions for  developing  attribute  weights. 

When  weights  have  been  determined  for  each  of  the  attributes,  one 
may  determine  a score  for  each  technique  bv  computing  the  weighted  sum 
of  the  deviations  of  each  reference  profile  from  the  situation  profile. 

For  each  technique  being  considered,  determine  the  deviation  from 
the  situation  profile,  multiply  each  of  these  deviations  by  the  appro- 
priate attribute  weight,  and  finally  sum  these  across  all  the  attributes 
of  each  technique. 

A deviation  is  considered  positive  if  the  technique  is  rated  higher 
than  the  situation  profile.  A deviation  is  considered  negative  if  the 
technique  is  rated  lower  than  the  situation  profile. 

Table  2 shows  an  example  of  these  calculations  for  one  technique. 

The  technique  that  has  the  highest  score  is  chosen  since  this  result 
indicates  the  least  undesirable  technique. 
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Table 

2.  Sample 

Calculations 

for  Weighted 

Aggregate  Approach 

Method 

Profile  Ratings 

Attribu tes 

Situation 

Technique 

’A"  Deviation  Weight 

Product 

Assessment 

Time 

80 

90 

+10 

10 

100 

Training 

Time 

80 

75 

-5 

3 

-15 

Flexibility 

25 

25 

0 

1 

0 

Face 

Validity 

55 

20 

-35 

2 

70 

Complexity 

75 

95 

+20 

9 

180 

Accuracy 

70 

75 

+5 

6 

30 

SCORE  FOR  TECHNIQUE  'A' 

+225 

DISCUSSION  AND  SUMMARY 

The  strategy  for  selecting  a worth  assessment  technique  structures 
the  analysis  of  an  assessment  problem  in  terms  of  ten  composite  attri- 
butes. The  underlying  assumption  is  that  certain  factors  of  the  total 
assessment  problem  are  critical  to  success  of  the  worth  assessment 
process . 

The  10  attributes  of  the  strategy  represent  a synthesis  of  41 
factors  identified  in  the  Delphi  procedure  (Figure  2).  The  success  of 
the  strategy  depends  on  the  extent  to  which  the  critical  factors  were 
captured  in  the  Delphi  and  incorporated  into  the  analysis. 

A relatively  small  number  of  those  queried  responded  to  the  Delphi 
questionnaire.  However,  even  the  small  sample  of  respondents  in  round  3 
(9)  represented  many  years  of  experience  in  worth  assessment  by  active 
practitioners  in  the  field. 
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The  nine  techniques  Included  In  the  analysis  (Figure  1)  represent 
the  most  frequently  used  assessment  techniques.  Each  of  the  techniques 
actually  represents  a family  of  techniques  with  similar  characteristics 
(Kneppreth  et  al,  1974,  op.  cit.).  Considering  additional  techniques 
would  not  require  a change  in  the  strategy,  but  would  require  extending 
the  procedure  to  develop  further  reference  profiles. 

The  10  attributes  of  the  techniques  represent  two  types  of  factors 
in  an  assessment  problem.  The  first  type,  the  four  dichotomous  attri- 
butes summarized  in  Table  1,  focuses  on  the  characteristics  of  the 
decision  attributes  in  the  assessment  problem. 

The  analysis  of  the  dichotomous  attributes  is  in  terms  of  the 
limitations  inherent  in  the  application  of  a worth  assessment  technique 
in  any  situation:  Is  the  technique  feasible  for  a specific  problem? 

The  second  type,  the  six  continuous  attributes,  focuses  on  the 
setting  of  the  worth  assessment  problem.  The  analysis  of  the  continuous 
attributes  is  in  terms  of  the  constraints  governing  the  application  of 
any  worth  assessment  technique  in  a situation:  Is  the  technique  accept- 
able for  a specific  situation? 

The  10  attributes  do  not  represent  an  exhaustive  synthesis  of 
factors  determining  the  successful  application  of  a worth  assessment 
technique.  The  goal  was  not  completeness,  but  rather  inclusion  of  the 
critical  factors. 

This  strategy  represents  an  initial  attempt  to  structure  the  prob- 
lem of  selection  of  a worth  assessment  technique.  The  strategy  is 
designed  to  guide  the  user  through  an  analysis  of  the  assessment  problem 
in  terms  of  factors  critical  in  the  successful  application  of  worth 
assessment . 

The  relative  contribution  of  the  different  factors  to  assessment 
success,  as  well  as  the  scales  used  for  depicting  situations  in  terms  of 
these  factors,  represents  the  authors'  analysis  and  insights  derived 
from  the  Delphi  process  and  a review  of  worth  assessment  techniques 
rather  than  being  empirically  derived. 

Further  definition  of  the  relationships  noted  in  the  analysis, 
validation,  and  refinement  of  the  strategy  is  required.  A strategy 
that  focuses  analysis  of  the  assessment  problem  on  factors  accepted  by 
experts  as  being  important  in  applying  worth  assessment  techniques  should 
increase  the  likelihood  of  successful  decisionmaking. 
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CONSTANT  PROBABILITY  WAGERS  involve  constructing  wagers  (usuallv 

with  probabilities  of  1/2)  and  then  varying  the  factor  levels 
until  the  dm  is  indifferent  between  the  wager  and  an  alterna- 
tive 'sure  thing'.  Yields  interval  data. 

VARIABLE  PROBABILITY  WAGF.RS  involve  constructing  wagers  and  then 

varying  the  probabilities  until  the  dm  is  indifferent  between 
the  wager  and  an  alternative  'sure  thing'.  Yields  interval 
data. 


MULTIVARIATE  METHODS 

INDIFFERENCE  CURVE  METHOD  involves  the  joint  assessment  of  two 

attributes  by  constructing  a plane  of  the  possible  combinations 
and  then  determining  indifference  points  and  curves . Can  be  used 
to  assess  two  dependent  or  two  independent  attributes  at  a time. 
Yields  ordinal  data. 


ORDINAL  METHODS 

RANKING  involves  placing  rank  numbers  on  a form  or  sorting  cards 
into  an  order.  Yields  ordinal  data. 

PAIRED  COMPARISONS  involve  choosing  the  preferred  attribute  levels 
in  each  of  a series  of  pairs.  Yields  ordinal  data. 

EQUIVALENCE  GROUPING  involves  placing  factor  levels  into  a discrete 

number  of  classes  representing  varying  degrees  of  worth.  It  may 
involve  forms  or  sorting  cards  into  piles.  Yields  partial 
rank  order  with  some  interval  information. 

DIRECT  METHODS 

SCALED  RESPONSE  (DOUBLE  ANCHOR)  involves  identifying  the  two 

extremes  of  the  attribute  scale  and  then  anchoring  the  extremes. 
Intermediate  attribute  levels  are  then  assessed  by  giving 
numbers  or  drawing  marks  on  a scale.  Yields  interval  data. 

SCALED  RESPONSE  (SINGLE  ANCHOR)  involves  anchoring  one  attribute 

level  to  a scale  and  then  rating  the  other  levels  in  relation 
to  this  point  (usually  by  ratios).  May  involve  eliciting 
numbers  or  marking  a scale.  Yields  interval  data. 

GRAPHICAL  METHOD  involves  drawing  directly  onto  a graph,  with  one 
axis  for  worth  and  the  other  for  the  attribute  levels,  a line 
which  represents  the  worth  function  of  the  attribute.  Some 
points  are  usually  anchored.  Yields  interval  data. 


Figure  1.  Brief  Descriptions  of  Nine  Worth  Assessment  Techniques 
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Relative 

Weight  Factor 

Decision  Problem/Si .uatlon  and  Purpose 

100  The  required  sensitivity  of  the  results 

92  The  decision  model 

‘6  The  size  of  the  decision  problem 

48  Whether  the  problem  involves  risk  or  not 

41  Whether  the  worths  are  invariant  over  time 

22  Whether  the  situation  is  static  or  dynamic 

18  Whether  the  purpose  is  normative  or  descriptive 

16  The  degree  of  accuracy  desired  in  the  results 

14  The  number  of  decisionmakers  to  be  assessed 

U Whether  past  choice  data  is  available 

2 Whether  one  is  modelling  choices  or  processes 

1 Whether  alternatives  can  be  compared  to  a standard 

0 Probability  of  a unique  final  choice 

0 The  order  of  processing  the  attributes 

0 The  location  of  the  worth  assessment  session 

Assessor /Analyst 

95  The  expertise  of  the  assessor 

39  The  amount  of  faith  that  the  assessor/analyst  has  in  the  technique 

14  The  assessor's  familiarity  with  the  technique 

6 Previous  relationship  between  the  assessor  and  decisionmaker  DM'* 

4 Motivation  of  the  assessor 

Decision  Attributes 

100  The  number  of  decision  attributes 

93  Whether  or  not  the  decision  attributes  are  independent 

15  Whether  attributes  are  nonmonotonically  related  to  worth 

14  Whether  attributes  are  continuous  or  discrete  and  have  definite  end  points 

7 Whether  attributes  can  be  resolved  into  a single  dimension 

2 Whether  some  combinations  of  attribute  levels  are  not  possible 

Declsionmaker( s) 

75  The  ease  with  which  the  DM  can  understand  the  technique 

70  The  importance  of  che  decision  to  the  decisionmaker 

45  The  Information  demanded  from  the  decisionmaker 

28  The  value  of  time  expended  by  the  decisionmaker 

23  Whether  the  DM  will  benefit  from  giving  his/her  worths 

1 The  sophistication  and  vocabulary  of  the  DM 

1 (For  Group  Decisions)  Whether  the  DM  will  feel  that  his/her  expertise  will  be  recognized 

0 (For  Group  Decisions)  Personality,  attitude,  and  sta  us  of  the  DM 

Techniques 

56  Known  biases  of  the  technique 

45  The  cost  of  administering  the  technique 

29  The  face  validity  of  the  technique 

21  The  amount  of  education  required  to  use  the  technique 

21  Whether  or  not  the  technique  allows  feedback 

10  Whether  the  results  of  the  technique  can  be  cross-checked 

6 Whether  the  technique  allows  inmediate  value  changing  or  experimenting 

Figure  2.  The  Relative  Weights  of  Critical  Factors  in  Worth  Assessment 
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Dichotomous  Attributes 


Independence : A decision  attribute  is  independent  if  its  value 
does  not  depend  on  knowing  the  level  of  another  attribute. 

Risk:  A decision  problem  Involves  risk  if  the  decision  is 
related  to  probabilistic  outcomes. 

Monotonicity:  A decision  attribute  is  monotonically  related 
to  value  if  an  Increase  (or  decrease)  in  the  attribute 
increases  (or  decreases)  the  value  to  the  DM  over  its  entire 
range . 

Discreteness:  A decision  attribute  is  discrete  if  only  a 
limited  number  of  levels  exist. 


Continuous  Attributes 


Assessment  Time:  The  relative  amount  of  time  required  to 
complete  an  assessment  of  a given  attribute  with  a DM  trained 
in  using  the  technique. 

Training  Time:  The  relative  amount  of  time  required  to  train 
a DM,  with  no  previous  knowledge  of  the  technique,  in  the  use 
of  the  technique. 

Face  Validity:  The  relative  appearance  to  the  DM  that  his/her 
responses  yield  useful  and  meaningful  information. 

Accuracy : The  relative  precision  with  which  the  technique 
elicits  a DM* s true  worth  perspective  as  well  as  the  technique's 
sensitivity  to  variations  in  the  DM's  worth  function. 

Flexibility:  The  relative  ease  with  which  the  technique  can 
be  modified  to^ suit  different  situations  and  DMs,  and  the 
ease  with  which  a DM  can  modify  his/her  worth  function. 

Complexity : The  relative  complexity  of  the  DM's  task  in  using 
the  technique;  including  the  abstractness  of  required  responses 
and  the  number  and  order  of  different  activities. 


Figure  3.  Attributes  Used  in  the  Selection  Strategy 
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Figure  4.  Flow  Chart  Summarizing  the  Strategy  for  Selecting  a Worth  Assessment  Technique 
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CONSTANT  PROBABILITY  WAGERS 

These  techniques  involve  constructing  wagers  (usually  with  proba- 
bilities of  1/2)  and  then  varying  the  factor  levels  until  the  dm  is 
indifferent  between  the  wager  and  an  alternative  'sure  thing'.  The 
technique  yields  interval  data. 


DICHOTOMOUS  FACTORS: 


Risk?  The  technique  involves  risk  and  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  While  the  technique  can  theoretically  handle 
nonmonotonically  related  attributes,  there  may  be  problems 
in  varying  the  factor  levels  and  constructing  wager9. 

Discrete?  The  technique  will  not  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 


The  crosshatched  portion  extends  one  standard  deviation  from 
the  mean  (shaded  portion) . 
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TIME 
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Figure  6.  Reference  Profile  of  Constant  Probability  Wagers 
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VARIABLE  PROBABILITY  WAGERS 

This  technique  involves  constructing  wagers  and  then  varying  the 
probabilities  until  the  dm  is  indifferent  between  the  wager  and  an 
alternative  'sure  thing'.  The  technique  yields  interval  data. 

DICHOTOMOUS  FACTORS: 

Risk?  The  technique  involves  risk  and  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  The  technique  can  theoretically  handle 

attributes  that  are  nonmonotonically  related  to  worth.  However, 
problems  may  arise  in  constructing  the  wagers. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from 
the  mean  (shaded  portion) . 
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Figure  7.  Reference  Profile  of  Variable  Probability  Wagers 
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INDIFFERENCE  CURVE  METHOD 

This  method  involves  the  joint  assessment  of  two  attributes,  and 
is  accomplished  by  constructing  a plane  of  the  possible  combinations 
of  the  two  attributes  and  then  determining  indifference  points  and 
curves  on  this  plane.  The  technique  yields  ordinal  data. 

DICHOTOMOUS  FACTORS: 


Risk?  This  technique  does  not  involve  risk  or  probabilities. 

Dependence?  This  technique  handles  dependencies  however,  it  is 
realistically  limited  to  assessment  of  only  two  dependent 
attributes. 

Nonmonotonicity?  This  technique  cannot  assess  attributes  which 
are  nonmonotonically  related  to  worth. 

Discrete?  This  technique  is  inappropriate  for  discrete  attributes. 

CONTINUOUS  FACTORS: 


The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean  (shaded  portion) . 
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Figure  8.  Reference  Profile  of  the  Indifference  Curve  Method 
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RANKING  METHODS 

These  techniques  involve  placing  rank  numbers  on  a form  or  sorting 
cards  into  rank  order.  The  technique  yields  ordinal  data. 

DICHOTOMOUS  FACTORS: 

Risk?  These  techniques  do  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  The  techniques  can  assess  attributes  which  are 
nonmonotonically  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean  (shaded  portion). 
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Figure  9.  Reference  Profile  of  Ranking  Methods 
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PAIRED  COMPARISONS 

This  technique  Involves  choosing  the  preferred  attribute  levels 
in  each  of  a series  of  pairs.  The  technique  yields  ordinal  data. 

D I CHOTOMOUS  FACTORS : 

Risk?  The  technique  does  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicitv?  The  technique  can  assess  attributes  which 
are  nonmonotonically  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean  (shaded  portion). 
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Figure  10.  Reference  Profile  of  Method  of  Paired  Comparisons 
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These  techniques  involve  placing  factor  levels  into  a discrete 
number  of  classes  representing  varying  degrees  of  worth.  They  may 
involve  forms  or  sorting  cards  into  piles.  The  technique  yields 
partial  rank  order  with  some  interval  information. 

DICHOTOMOUS  FACTORS: 

Risk?  The  technique  does  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  The  technique  can  handle  attributes  which  are 
nonmonotonically  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from 
the  mean  (shaded  portion) . 
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Figure  1 1 . Reference  Profile  of  Equivalence  Grouping 
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SCALED  RESPONSE  (DOUBLE  ANCHOR)  METHOD 

The  technique  involves  anchoring  the  extremes  of  the  worth  scale 
to  the  extreme  attribute  levels  and  then  assessing  intermediate  levels 
in  relation  to  these  two  points  either  by  giving  numbers  or  drawing 
marks  on  a scale.  The  technique  yields  interval  data. 

DICHOTOMOUS  FACTORS: 

Risk?  The  technique  does  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  The  technique  will  handle  attributes  which 
are  nonmonotonically  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from 
the  mean  (shaded  portion). 
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Figure  12.  Reference  Profile  of  Scaled  Response  (Double  Anchor)  Method 
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This  technique  involves  anchoring  one  attribute  level  to  a scale  and 
then  rating  the  other  levels  in  relation  to  this  point  (usually  by 
ratios).  May  involve  eliciting  numbers  or  marking  on  a scale.  The 
technique  yields  interval  data. 

DICHOTOMOUS  FACTORS: 

Risk?  The  technique  does  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicitv?  The  technique  will  handle  attributes  which  are 
nonmonotonically  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean  (shaded  portion) . 
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Figure  13.  Reference  Profile  of  Scaled  Response  (Single  Anchor) 
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GRAPHIC  METHOD 


This  technique  involves  drawing  directly  onto  a graph,  with  one  axis 
for  worth  and  the  other  for  the  attribute  levels,  a line  which  represents 
the  worth  function  of  the  attribute.  Some  points  are  usually  anchored. 
The  technique  yields  interval  data. 

DICHOTOMOUS  FACTORS: 

Risk?  The  technique  does  not  involve  risk  or  probabilities. 

Dependence?  The  technique  will  not  handle  dependent  attributes. 

Nonmonotonicity?  The  technique  will  handle  attributes  which  are 
nonmonotonicallv  related  to  worth. 

Discrete?  The  technique  will  handle  discrete  attributes. 

CONTINUOUS  FACTORS: 

The  crosshatched  portion  extends  one  standard  deviation  from  the 
mean  (shaded  portion) . 
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Figure  14.  Reference  Profile  of  Graphic  Method 
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Read  the  list  of  attributes  below  and  choose  the  one  that  you  feel 
is  least  important  in  your  decision  situation.  Assign  a score  of  1 to 
this  attribute.  Now  choose  the  one  which  seems  to  be  most  important  and 
assign  a 10  to  it.  These  two  numbers  form  the  endpoints  of  a scale  of 
importance.  Please  rate  the  other  four  attributes  in  relation  to  the 
two  already  specified.  These  numbers  become  the  weights  for  the  attri- 
butes for  part  3 of  the  strategy. 


SCORE  ATTRIBUTE 

Assessment  Time  - the  amount  of  time  available  for  assessment 

sessions . 

Training  Time  - amount  of  time  available  for  training  the  dm. 

Face  Validity  - the  amount  of  faith  the  dm  must  have  in  the 

method. 

Accuracy  of  Results  - the  required  specificity  of  the  results. 

Flexibility  - the  number  of  variations  required. 

Complexity  - difficulty  of  the  dm's  task. 


Figure  15.  Instructions  for  Weighting  the  Method  Attributes 
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APPENDIX  A 


EVALUATION  QUESTIONNAIRES  FOR  ASSESSING  SITUATIONAL  REQUIREMENTS 
FOR  ASSESSMENT  TIME,  TRAINING  TIME,  FACE  VALIDITY, 
ACCURACY,  FLEXIBILITY,  AND  COMPLEXITY 

ASSESSMENT  TIME 


Score 


Time  Pressure:  Rate  on  a zero  to  20  scale  the  relative  time 

horizon  of  the  decision  process.  Rate  as  zero  a situation 
in  which  there  is  no  time  pressure  to  make  a decision. 

Rate  as  20  a situation  in  which  the  decision  must  be  made 
as  quickly  as  possible  

Decisionmaker  (DM)  Time  Available:  Rate  on  a zero  to  30  scale 
the  amount  of  DM  time  available.  Rate  as  zero  when  there 
is  an  unlimited  amount  of  DM  time.  Rate  as  30  when  there 
is  little  time  available.  Time  available  will  depend  not 
only  on  the  physical  constraints  of  the  DM  but  also  on 
the  DM's  motivation.  If  the  DM  is  unmotivated,  he/she  may 
not  be  willing  to  give  much  time  to  the  assessment  .... 

Number  of  Decisionmakers:  Rate  on  a zero  to  20  scale  the  number 
of  DMs  to  be  used  in  the  assessment  process.  Rate  as  zero 
when  there  is  only  one  DM.  Rate  as  20  when  there  are  five 
or  more  DMs.  If  more  than  one  DM  are  going  to  be  treated 
as  a group  and  there  will  be  one  assessment  session  for 
the  group,  rate  this  as  a 5 

Number  of  Attributes:  Rate  on  a zero  to  J>0  scale  the  number  of 
attributes  to  be  assessed.  Rate  as  zero  when  there  is 
only  one  attribute  to  be  assessed.  Rate  as  30  when  there 
are  10  or  more  attributes  to  be  assessed  


TOTAL  SCORE  (sum  of  the  above) 


Other  Considerations:  The  total  score  can  be  modified  slightly  depending 
on  several  considerations.  For  instance,  if  a decisionmaker  is  not 
sophisticated  and  likely  to  be  slow  with  worth  assessment,  then 
increase  the  total  score.  Or,  if  there  is  little  money  to  pay  the 
DMs  for  their  time  (if  this  is  the  case) , then  the  total  score  also 
should  be  increased.  If  any  other  characteristics  of  the  situation 
tend  to  make  time  critical,  increase  the  total  score  and  if  any 
characteristic  makes  time  less  critical,  decrease  the  total  score. 


A-l 


Time  Pressure:  Rate  on  a zero  to  20  scale  the  relative  time 
horizon  of  the  decision  process.  Rate  as  zero  a 
situation  in  which  there  is  no  time  pressure.  Rate  as 
20  a situation  in  which  the  decision  must  be  made  as 
quickly  as  possible  


Decisionmaker  Time  Available:  Rate  on  a zero  to  20  scale  the 
amount  of  time  that  the  DMs  have  available.  Rate  as  zero 
a situation  where  the  DMs  have  an  unlimited  amount  of 
time  to  spend.  Rate  as  20  a situation  where  the  DMs  have 
very  little  time  available.  If  the  DMs  are  very  unmoti- 
vated, training  may  take  more  time  and  the  value  here 
should  be  increased  


Number  of  Decisionmakers:  Rate  on  a zero  to  50  scale  the  relative 
number  of  decisionmakers.  Rate  as  zero  a situation  where 
only  one  DM  need  be  trained.  Rate  as  30  where  five  or 
more  DMs  are  to  be  trained.  If  the  DMs  are  to  be  trained 
as  a group  then  rate  this  factor  a 10 


Number  of  Methods:  Rate  on  a zero  to  50  scale  the  relative 

number  of  methods  that  will  be  employed  in  the  assessmen 
process.  This  is  a measure  of  the  number  of  techniques 
be  employed  for  the  different  types  of  attributes.  Rate 
as  zero  a situation  where  only  one  technique  will  be 
used.  Rate  as  50  a situation  where  four  or  more  tech- 
niques will  be  used  


t 

to 


TOTAL  SCORE  (sum  of  the  above) 


Other  Considerations:  The  total  score  can  be  modified  based 

on  several  other  considerations.  If  the  DM  is  not  familiar  at 
all  with  worth  assessment  or  if  he/she  is  not  sophisticated  in 
these  types  of  techniques,  training  may  take  longer  and  the 
value  of  the  total  score  should  be  increased.  Tf  there  are 
factors  which,  in  your  situation,  will  decrease  the  time  for 
training,  decrease  the  total  score  appropriately. 
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FACE  VALIDITY 


Past  Contact  Between  DM  and  Assessor:  Rate  on  a zero  to  50 

scale  the  relative  degree  of  faith  that  the  DM  has  in  the 
assessor.  It  will  depend  on  past  contacts  with  the  DM, 
and  the  reputation  of  the  assessor  as  perceived  by  the  DM. 
Rate  as  zero  a situation  in  which  the  DM  has  complete 
faith  in  the  assessor.  Rate  as  50  a situation  in  which  the 
DM  distrusts  the  assessor.  Rate  as  20  if  the  DM  and  the 
assessor  have  not  met  

Importance  of  the  Decision  to  the  DM:  Rate  on  a zero  to  30  scale 
the  relative  importance  of  the  decision  to  the  DM.  Rate 
as  zero  a situation  in  which  the  DM  does  not  consider  the 
decision  important.  Rate  as  30  a situation  in  which  the  DM 
feels  that  the  decision  is  very  important  and  wants  to 
ensure  that  his/her  value  is  assessed  validly  

Status  of  the  DM:  Rate  on  a zero  to  20  scale  the  relative  status 
of  the  DM.  Rate  as  zero  a situation  in  which  the  DM  is  of 
low  status  and  has  been  "ordered"  to  take  part  in  the 
assessment.  Rate  as  20  a situation  in  which  a high  status 
person  who  is  taking  part  in  the  assessment  of  his/her  own 
free  will  


TOTAL  SCORE  (sum  of  the  above) 


Other  Considerations:  The  education  and  sophistication  of  the 
DM  might  affect  this  factor.  A highly  sophisticated  DM 
may  require  the  technique  to  have  a higher  face  validity 
(or  possibly  a lower  face  validity).  In  any  case  if  it 
is  felt  that  a higher  face  validity  is  needed  for  one 
reason  or  another,  the  total  score  may  be  Increased. 
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ACCURACY 


Score 

Cost  of  Making  an  Incorrect  Decision:  Rate  on  a zero  to  30 
scale  the  relative  importance  of  the  decision.  Rate  as 
zero  a situation  in  which  the  decision  is  not  important 
and  no  adverse  effects  would  occur  from  making  an  incorrect 
decision.  Rate  as  30  a situation  in  which  the  decision  is 
extremely  critical  and  the  cost  of  making  an  incorrect 
decision  is  very  high 


The  Model:  Rate  on  a zero  to  20  scale  the  type  of  measure- 
ments required  by  the  decision  model  to  be  used.  Rate  as 
zero  a model  which  requires  anything  less  than  full  rank 
order,  and  rate  as  20  a model  which  requires  strict 
interval  data  


Number  of  DMs:  Rate  on  a zero  to  30  scale  the  number  of  DMs 
to  be  used  in  the  assessment  process.  If  there  are  many 
decisionmakers  whose  worth  functions  will  somehow  be 
averaged,  then  the  individual  assessments  will  be  less 
critical.  Rate  as  zero  a situation  in  which  there  are  many 
DMs  over  which  worths  would  be  averaged.  Rate  as  30  a 
situation  in  which  there  is  only  one  DM  


Number  of  Attributes:  Rate  on  a zero  to  20  scale  the  number 
of  activities  to  be  assessed.  As  the  decision  is  based 
on  more  attributes,  the  effects  of  any  single  one  will  be 
reduced.  Rate  as  zero  a situation  in  which  there  are  very 
many  attributes.  Rate  as  20  a situation  where  there  are 
few  attributes ' 


TOTAL  SCORE  (sum  of  the  above) 


Other  Considerations:  There  are  many  other  considerations  which 
will  affect  the  required  sensitivity  of  the  worth  assessment 
technique.  For  instance,  if  the  purpose  of  the  assessment 
is  to  determine  how  values  change  over  time,  it  may  be 
important  to  use  a more  sensitive  method.  If  the  situation 
requires  a more  accurate  method,  increase  the  total  score. 
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The 


Number  of  Different  Variations:  Rate  on  a zero  to  40 
scale,  the  number  of  different  variations  of  assessment 
techniques  that  will  be  required  for  different  decision 
attributes.  This  is  an  indication  of  the  number  of  differ- 
ent types  of  attributes  that  the  method  will  encounter. 

Rate  as  zero  a situation  where  the  technique  will  not  have 
to  be  modified  to  suit  different  attributes  


The  Number  of  Different  DMs:  Rate  on  a zero  to  60  scale  the 
number  of  variations  of  the  selected  method  that  will  be 
required  to  suit  different  decisionmakers.  Rate  as  zero  a 
situation  in  which  no  variations  will  be  necessary.  Rate  as 
60  a situation  where  there  are  five  or  more  different  DMs 
who  will  require  variations  


Score 


TOTAL  SCORE  (sum  of  the  above) 


Other  Considerations : In  general  this  will  not  be  an 
important  consideration  in  the  selection  of  a worth 
assessment  technique.  However,  in  some  situations  it  is 
important  to  tailor  the  technique  to  meet  various  conditions. 
In  this  type  of  situation  the  total  score  should  be  increased. 
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COMPLEXITY 


Score 


Sophistication  of  the  DM:  Rate  on  a zero  to  50  scale  the 
educational  level  and  quantitative  abilities  of  the  DM. 
Rate  as  zero  a situation  in  which  the  DM  is  highly  educa- 
ted and  unlikely  to  have  trouble  with  any  of  the  worth 
assessment  techniques.  Rate  as  50  a situation  in  which 
the  education  level  is  low  and  there  is  a good  possi- 
bility that  the  DM  will  have  much  trouble  

Motivation  of  the  DM:  Rate  on  a zero  to  50  scale  the 

relative  level  of  motivation  of  the  DM.  Rate  as  zero  a 
situation  in  which  the  DM  feels  the  decision  is  important 
and  is  highly  motivated.  Rate  as  50  a situation  in  which 
the  DM  does  not  feel  the  decision  is  important  and  is  not 
motivated  to  cooperate  with  the  assessment 


TOTAL  SCORE  (sum  of  the  above)  . . . 
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